IMAGE FILTER METHOD 



CROSS-REFERENCE TO RELATED APPLICATIONS 

The following patent applications disclose related subject matter: Appl. 

Nos. 09/ , filed .... ( — ). These referenced applications have a common 

assignee with the present application. 

BACKGROUND OF THE INVENTION 

The invention relates to image processing, and more particularly to image 
filtering methods and related devices such as digital and video cameras. 

There has been considerable growth in the sale and use of digital 
cameras, both still and video, in recent years. Figure 4 is a block diagram of a 
typical digital still camera which includes various image processing components, 
collectively referred to as an image pipeline. Color filter array (CFA) 
interpolation, gamma correction, white balancing, color space conversion, and 
JPEG (or MPEG for video) compression-decompression constitute some of the 
key image pipeline processes. The typical color CCD consists of a rectangular 
array of photosites (pixels) with each photosite covered by a color filter: e.g., red, 
green, or blue. Thus to recover a full-color image (all three colors at each pixel), 
a method is required to calculate or interpolate values of the missing colors at a 
pixel from the colors of its neighboring pixels. Such methods lead to artifacts in 
the full color image which should be removed by filtering. 

Similarly, high-frequency elements of video images are commonly 
eliminated using conventional linear filters. However, the size of such filters 
becomes large when the desired characteristics are demanding, and this results 
in prohibitively large circuit size. 

Infinite impulse resonse (IIR) filtering is often used in acoustical signal 
processing. However, it is little used in image processing due to its side effects, 
which are often imperceptible in sound but apparent in images. 

Filtering using the matching method compares input signals with a stored 
database and outputs appropriate signals. Although this method works well in 
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some situations, the output quality can be low if the database does not match the 
input. Also, this method consumes large amounts of memory and computational 
power. 

SUIVIMARY OF THE INVENTION 

The present invention provides image processing methods and systems 
with filtering using estimates of edge intensity and direction to smooth parallel to 
an edge. 

This has advantages including enhanced quality of image. 

BRIEF DESCRIPTION OF THE DRAWINGS 
The drawings are heuristic for clarity. 
Figure 1 is a flow diagram for a preferred embodiment method. 
Figures 2a-2b illustrate edges. 
Figures 3a-3b show experimental results. 
Figure 4 is a block diagram of a digital camera system. 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 



1 . Overview 

Preferred embodiment image filtering methods include defining a non- 
linear filter by first determining at each pixel any neighboring edge direction and 
strength, and then defining filter coefficients for such pixel by roughly rotating a 
one-dimensional smoothing filter to align with the edge. That is, the filter 
coefficients adapt to the pixel's edge environment. Filtering the image with this 
pixel-dependent filter preserves edges and smoothes noise and artifacts 
paralleling edges. Figure 1 is a flow diagram. The methods allow relatively small 
filters (e.g., 5x5) and simple computations to provide high quality output; this 
permits use of small buffer memory together with low MIPS on a programmable 
processor and/or simple hardwired circuitry. 

Preferred embodiment digital image processing systems (such as 
cameras) include preferred embodiment image filtering methods. 

Figure 4 shows in functional block form a system (camera) which may 
incorporate preferred embodiment methods, such as in the edge detection block. 
The functions of preferred embodiment systems can be performed with digital 
signal processors (DSPs) or general purpose programmable processors or 
application specific circuitry or systems on a chip such as both a DSP and RISC 
processor on the same chip with the RISC processor as controller. Further 
specialized accelerators, such as CFA color interpolation and JPEG encoding, 
could be added to a chip with a DSP and a RISC processor. Captured images 
could be stored in memory either prior to or after image pipeline processing. The 
image pipeline functions could be a stored program in an onboard or external 
ROM, flash EEPROM, or ferroelectric RAM for any programmable processors. 

2. First preferred embodiment 

Figure 1 is a flow diagram of first preferred embodiment filtering methods 
which include the following steps to define filter coefficients for filtering an image. 
Let p(n,m) denote the pixel value at pixel location (n,m) in the image: the first 
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index indicates the column measured from left to right and the second index 
indicates the row measured from bottom to top. Note that p(n,m) could represent 
luminance, chrominance, red, green, blue, magenta, etc. 
(1 ) First, for location (n,m) estimate the direction and intensity of any nearby 
edges. In particular, consider 5x5 neighborhoods of pixels for edge estimations, 
and compute horizontal (dx) and vertical (dy) variations at (n,m) as follows. 
dx(n,m) = S-2<j^i,-2<k<2 aO.k) |p(n+j,m+k) - p(n+1+j,m+k)| 
dy(n.m) = Z-2<j<2, -2<k£i a(k,j) |p(n+j,m+k) - p(n+j,m+1+k)| 
with the coefficients a(j,k) as: 

r 



aC,k) = 



1 2 2 

2 4 4 
4 8 8 
2 4 4 
1 2 2 



where the lower left coefficient is a(-2,-2) and the upper right coefficient is 
a(1 ,2), Thus dx(n,m) measures the variation of p(.,.) in the x direction about (n,m) 
and dy(n,m) measures the variation in the y direction. If there is an edge at (n,m), 
then the discontinuities in p(.,.) across the edge dominate the sums. Thus define 
the edge intensity at (n.m), l(n,m), to be the maximum of dx(n,m) and dy(n,m), 
and estimate the edge angle, 0, as illustrated in Figures 2a-2b, by ±arctan(dx/dy). 

To account for the sign of 6, two diagonal variation measures for the same 
5x5 neighborhood of (n,m) are defined: di (measures variation across edges as in 
Figure 2a) and dr (measures variation across edges as in Figure 2b), as follows: 
di(n,m) = I -2<j^i,-2<kii b(j,k) |p(n+1+j,m+k) - p(n+j,m+1+k)| 
cir(n,m) = I -2sj^i,-2<k^i b(j,k) |p(n+j,rn+k) — p(n+1+j,m+1+k)| 
where the coefficients bQ.k) are: 

12 2 1" 



bO.k) = 



2 4 4 2 
2 4 4 2 
12 2 1 
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where the lower left coefficient Is b(-2 -2) and the upper right coefficient is 

b(1 ,1 ). Then estimate the sign of 9 as positive if di > dr and negative if di < dr. 

(2) The foregoing computations dx(n,m), dy(n,m), di(n,m), and dr(n,m) provide 

estimations of edge angle and intensity and are used to create the 5x5 filter 

matrix to apply to the image at (n,m). Essentially, the 5x5 filter matrix will be an 

approximate rotation of the following 5x5 x-direction-smoothing filter which 

preserves horizontal edges: 

"0 0. 0 0 0" 
0 0 0 0 0 
M = (1/16) 2 4 4 4 2 
0 0 0 0 0 
0 0 0 0 0 

Note that M would be used unchanged for a pixel at an edge with slope 6=0. 

First, the range of 6 is partitioned into 8 intervals, and each interval has a 
5x5 filter matrix which approximates a rotation of M by essentially weighting the 
matrix elements according to tanO of the element location. For convenience 
define r = dx/dy = tan|9|; then the 8 filter matrices are as follows: 

(a) For 0 < r < 1/2 and di > dr (corresponding to 0 < 0 < arctan(1/2) = 

26.6°), the 5x5 filter matrix is: 

^ 0 0 0 
0 0 0 
2(1 -2r) 4(1 -r) 4 
2(2r) 4r 0 
0 0 0 

where the [n,m] index denotes the matrix apples to the neighborhood centered at 
(n,m) and the (j,k) index denotes the location within the matrix with -2 <j < 2 and 
-2 < k < 2. 

(b) For 1/2 < r < 1 and di > dr (corresponding to 26.6° < 8 < 45°), the 5x5 
filter matrix is: 



M[n,m]G.k) = (1/16) 



0 0 

4r 2(2r) 

4(1 -r) 2(1 -2r) 

0 0 

0 0 
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M[n,m]a.k) = (1/16) 



0 
0 
0 

2(2 -2r) 
2(2r-l) 



0 
0 

4(1 -r) 
4r 
0 



0 0 
0 4r 

4 4(1 -r) 
0 0 
0 0 



2(2r-l) 
2(2 -2r) 

0 

0 

0 



(c) For 1 < r < 2 and di > dr (corresponding to 45° < 9 < 63.4°), the 5x5 filter 
matrix is the matrix of (b) with r replaced by 1/r and the matrix reflected across its 
lower-left to upper-right diagonal; that is: 



M[n.m]a.k) =(1/16) 



0 
0 
0 
0 



0 
0 
0. 
Air 



2(2/r-l) 2(2-2/r) 



0 2(2-2/r) 2(2/r-l) 
4(l-l/r) Air 0 

4 0 0 

4(1 -1/r) 0 0 

0 0 0 



(d) For 2 < r < 00 and di > dr (corresponding to 63.4° < 0 < 90°), the 5x5 
filter matrix is the matrix of (a) with r replaced by 1/r and the matrix reflected 
across its lower-left to upper-right diagonal. 

(e) For 0 < r < 1/2 and dr > di (corresponding to 0 > 6 > -arctan(1/2) = 

-26.6°), the 5x5 filter matrix is the matrix of (a) vertically reflected; that is: 

0 0 0 0 0 " 

2{2r) 4r 0 0 0 

2(1 -2r) 4(1 -r) 4 4(1 -r) 2(1 -2r) 

0 0 0 4r 2(2r) 

0 0 0 0 0 



M[n,m](j,k) =(1/16) 



(f) For 1/2 < r < 1 and dr ^ di (corresponding to -26.6° > 9 > -45°), the 5x5 
filter matrix Is the matrix of (b) vertically reflected. 

(g) For 1 < r < 2 and dr > di (corresponding to -45° > 9 > -63.4°), the 5x5 
filter matrix is the matrix of (f) with r replaced by 1/r and the matrix reflected 
across its lower-left to upper-right diagonal. 

(h) For 2 < r < 00 and dr > di (corresponding to -63.4° > 9 > -90°), the 5x5 
filter matrix is the matrix of (e) with r replaced by 1/r and the matrix reflected 
across its lower-left to upper-right diagonal. 
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(3) Apply the filter defined In step (2) with a weighting dependent upon the edge 
intensity to the image. In particular, first filter the input image pixel p(n,m) with 
the 5x5 filter M[n,m] computed in step (2) to yield q(n,m): 

q(n,m) = Z -2<j<2, -2<k<2 M[n,m]G,k) p(n+j,m+k) 
Then output the weighted sum of p(n,m) and q(n,m) as the new, filtered infiage 
p"®^(n,m), using the edge intensity l(n,m) computed in step (1): 

p"®^(n,m) = a l(n,m) q(n,m) + [1 - a l(n,m)] p(n,m) 
where a is a user-defined parameter reflecting the desired filtering strength and 
the size of l(n,m). When a l(n,m) is greater than 1 , saturation at a l(n,m) = 1 
could be used. For example, with p(n,m) representing luminance values in the 
range 0 to 255 and with an edge at p(n,m) having a discontinuity of about 100, 
l(n,m) may be on the order of 1000, and so a may be on the order of 0.001 . 

The relatively small 5x5 filter leads to two benefits: (i) the buffer memory 
containing filtered pixels not yet written back to image memory need only hold 
two rows plus two pixels, and (ii) the summations of steps (1)-(3) have at most 20 
nonzero terms. 



3. Alternative preferred embodiment 

An alternative preferred embodiment includes the case of RGB images. A 
section 2 filter can be applied to each of the R, G, and B elements separately. 
For the Y-Cb-Cr format, the filtering is not straight forward, because the 
resolution of the chroma is usually lower than that of the luminance. In the case 
of 4:2:0 video images, the a(j,k) and bG,k) matrices used in the chroma 
estimation of the slope can be replace with 

"1 2 r 



aG,k) = 



1 2 1 



and 



bO.k) = 

and the 5x5 filter matrix becomes a 3x3 fiter matrix: 



1 1 
1 1 
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M =(1/4) 



0 0 r 
1-r 2 l-r 
r 0 0 



4. Alternative smoothing preferred embodiment 

The preferred embodiment of section 2 could be modified to use different 
smoothlngs parallel to the edges. In particular, If the smoothing is [1,2,4,2,1] 
rather than [2,4,4,4,2], then the 5x5 matrix of case (a) would be: 

' 0 0 0 0 0 

0 0 0 

l-2r 2(l-r) 4 

2r 2r 0 

0 0 0 



M[n,m](j,k) =(1/10) 



2r 2r 

2(l-r) l-2r 
0 0 
0 0 



and analogously for cases (b)-(h). 



5. Experimental results 

Figure 3a shows black lines on white background with various slope 
angles, and Figure 3b shows the results of filtering with the preferred 
embodiment of section 2 pixel values in the range 0-255 and a = 0.02 (strong 
filtering). Note that the lines with slope angles of 45° and 90° are not affected by 
the filtering, whereas the bumps in the lines with angles between 45° and 90° are 
significantly smoothed. 

Because a line with slope angle 90° has no artifacts, the filtering ideally 
should have no effect, and the preferred embodiment is consistent with this. 

Artifacts in lines with slope angle of 45° are often not visible or not 
annoying to human perception. Thus these lines can be output without filtering; 
othenwise, clearness may be degraded. On the other hand, the lines with slope 
angles between 45° and 90° are smoothed by the filter Because the bumps in 
these lines are visible and often degrade image quality, ideally these bumps are 
reduced. Also, the edges of the lines are not too blurred by the preferred 
embodiment filtering, despite the smoothness. 
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6. Modifications 

The preferred embodiments may be modified in various ways while 
retaining one or more of the features of filtering derived from neighboring edge 
intensity and slope. 

For example, the 5x5 size of the neighborhood could be varied to 3x3 or 
7x7 or other (non-symmetrical) sizes. The partitioning of the angle into 8 
intervals could be replaced by a finer partitioning (more useful with larger filter 
neighborhoods) or coarser partitioning (and more nonzero filter matrix elements). 
The weighting matrices aG,k) and b(j,k) could be varied, and the filter matrix 
M[n,m](j,k) could vary and also adjust to the size of the neighborhood. 
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